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The Job Training Partnership Act is designed to be a performance- 
driven program. This means that the desired performance is stated in 
advance, clearly dsSning the goals of the progi-am. The performance 
standards system is one of the tools used to assure that job training 
program? are a productive investment in human capital. 

Since 1983, the U.S. Department of Labor has established national 
performance measures and set national numerical performance stan- 
dards to help assure that the goals of the Job Training Partnership Act 
are achieved. The U.S. Department of Labor has also designed a 
national adjustment methodology, sometimes called the "regression 
model," for adjusting the national standards based on local conditions. 
Each state must set standards for service deliveiy areas within that 
state using the Secretary's National standards, USDOL's adjustment 
methodology or its own adjustment methodology established within 
parameters set by the Secretary of Labor. States must also determine, 
on an annual basis, whether these performance standards have been 
met. They may also include State standards in their incentive poHcies, 
but the major portion of the incentive funds must be used for rev/ard- 
ing performance for the Secretary's measures. States must base their 
incentive award policies on achievement of these standards. Where 
standards have not been met by service delivery areas. States must 
provide technical assistance and may impose sanctions. Sanctions are 
actually required if an SD A fails to meet all performance standards for 
two consecutive years. Service delivery areas must formulate policies 
and operate programs that will meet or exceed the state-established 
performance standards, but they are encoura&jd to seek further 
adjustments if local conditions warrant. 

Clearly, the most challenging aspects of implementing the PY 90 
perforn'ance standards system fall to professionals at the local level. 
It is there that policies are made and programs are shaped. Tiie 
purpose of this guide is to provide local policy makers and program 
managers with an approach for using performance standards as a tool 
for reviewing and improving performance at the local level. The guide 
is keyed to the PY 90 DOL adjustment methodology for activities 
beginning July 1, 1990. 

Section I: Overview of Performance Standards — This section 
discusses the evolution of and rationale supporting performance 
standards. 

Section II: Identification of Local Performance — This section 
details a systematic approach to obtaining comprehensive baseline 
data on local performance, an analysis of the performance of specific 
activities and contractors within the SDA, and an analysis of perfor- 
mance trends. 



Section III: Strategies for Improving Performance — This sec- 
tion describes three approaches for improving performance: first, the 
Oversight Response, which identifies probable causes of poor perfor- 
mance; second, the Planning Response, which provides tools to exam- 
ine new strategies and approaches which will improve performance; 
and lastly, the follow-up response which reviews strategies related to 
the new postprogi'am measures. 

Section IV: Conclusion — This section summarizes the key issues 
related to planning and managing with performance standards. 

Readers are cautioned not to rely solely on performance standards to 
plan, manage, and evaluate JTPA; the performance standards system 
is one aspect of measuring the effectiveness of JTPA. The mission 
established by the private industry council (PIC) and local elected 
officials may make other goals as significant as achieving the perfor- 
mance standards. For example, the use of JTPA in effecting institu- 
tional change at the local level or the use of JTPA in promoting job 
creation and economic development activities could be as important as 
local perfoi-mance measures and must be considered in the self- 
evaluation process. 

Many of the reports and worksheets presented in the guide require 
routine, yet time-consuming computations using local participant 
data. A software option is available for microcomputer user? that 
speeds up the analysis. This program operates on MS DOS 2.0 and 
higher and requires an IBM PC, XT, AT or true compatible with a 
minimum of 256K The cost of the software is $199.00 with future year 
updates priced at $50.00. For more information on the software, 
contact the Employment & Training Institute at 1-800-932-0085. 

The .software option includes the following reports and worksheets 
which are referenced in this guide: 



Title 

Performance Status Report 

PY 90 Performance Standards Worksheet — Adult 

PY 90 Performance Standards Worksheet — Welfare . . . 
PY 90 Performance Standards Worksheet — Youth — 
PY 90 Perform^mce Standards Worksheet — Dislocated 



Worksheet 
Numher 



. 1 

13 
14 
15 
16 
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Section I: Overview of Performance Standards 



Throughout the 25 year evolution of Federal job training programs, 
government officials, local policy makers and job training practition- 
ers have agreed on the need to assess program performance. But two 
difficult issues have always had to be addressed: what are the best 
measures of effectiveness and wfiat is an equitable method for assess- 
ing performance? 

Measures of For a long time, the accepted wisdom in the employment and training 
Effectiveness field was that the importait measures of effectiveness concemed 
whether an adult obtained a job upon leaving the program, and what 
the cost of this job placement was. For youth, it has always been felt 
that acceptable results included placement into a job as well as other 
"positive" results, such as a retum to school or continued schooling. 
The system has traditionally measured the cost of these positive 
results for youth as well. Measures such as these were in use for many 
years, and served a useful purpose. They focused the employment and 
training system on the need to improve employability and get people 
jobs and to do so in a responsible fiscal manner. 

In Program Year 1986< July 1, 1986), the U.S. Department of Labor 
began to require the collection of data on what happened to adult 
participants after they left the program. Two years later, USDOL 
announced its intention to measure performance based on post- 
program data? Beginning with Program Year 1990 (July 1, 1990), the 
measures of efFectiveneso for adults will concern only what happened 
to them after they left the program. For youth, placements and other 
positive results are still considered the important indicators, but these 
have been carefully defined. 

Cost standards have been eliminated completely in order to move the 
JTPA system toward providing more intensive services to harder- to- 
serve individuals and in order to encourage more coordination with 
related pro.^ams. 

This direction in performance measures reflects the U.S. Department 
of Labor's national goals of long-term employability and economic 
self-sufficiency for the individuals enrolled in the Job Training 
Partnership Act. 

An equitable method for assessing performance of local programs all 
across the country is a complex issue. Each community has a unique 
set of circumstances which directly affects its ability to achieve 
positive results for individuals in that community. A program in an 
urban setting with a high unemployment rate serving a high percent- 
age of"hard-to-serve" individuals maybe performingwell if 50 percent 
ofits adult trainees getand keep jobs. A similar program in a suburban 
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Methods for Assessing 
Performance 




setting with a low unemployment rate serving a low percentage of 
"hard-to-serve" individuals may be performing poorly if only 50 
percent of its adult trainees get and keep jobs. Ten years ago, the 
expected performance of each program was determined by negotia- 
tions between local program operators and representative's of the U.S. 
Department of Labor. But there was no precision to this negotiation 
process; there was little data to support the various different perfor- 
mance goals which were negotiated. 

Then, 8 years ago, the U.S. Department of Labor introduced a new 
perforaiance standards methodology for JTPA. It is a regression- 
based statistical model which adjusts for factors found to have a 
significant effect on performance. The model has evolved over the 
years and now includes 29 variables which were found to have a 
significant impact on performance. Each variable is weighted accord- 
ing to its average impact on performance. For example, it was found 
that serving more dropout? resulted in fewer placements. The influ- 
ence of this factor was determined mathematically and a correspond- 
ing weight was assigned. The current methodology is an extension of 
this simple example. Thus, the performance standards systemincludes 
an "adjustment methodology" for determining the actual local stan- 
dard which reflects such factors as who is served and local economic 
conditions. Per example, an SDA v^th a low unemployment rate 
serving a low percentage of difficult-to-place trainees will have rela- 
tively high standards of performanc 3. In contrast, an SDA with a high 
unemploymem* rate serving a high percentage of difficult to-piace 
trainees will have relatively low standards of performance. If the 
model works as intended, the latter SDA would not be penalized 
because of difficult local conditions; its standards would be adjusted 
dov/nward. 

The specific performance standards and the variables used have 
changed over v.he 8 years , but the methodology — a na tional system for 
judging performance that allows adjustment for local factors — has 
remained constant. Many critical issues have been addressed in the 
establishment of this methodology. Many of these issues include 
technical items, such as statistical mode^ng, selection of databases 
from JTPA, and tolerance level • djustments. Three of the most 
significant policy issues addressed deal with the selection of the 
performance measures, the establishment of national standards, and 
the identification of the local adjustment factors. A description of each 
of these issues follows. 

Core Perfornance The Department of Labor has identified six core performance mea- 
Measures suresforProgram Years 1990 and 199 KJulyl, 1990— JuneSO, 1992) 
for use under Title IIA. Four of the six measures are for postprogram 
employment and earnings for adults and adult welfar e recipients. The 
remaining two measures focus on youth and measure employment 
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and employability development. Separate measures were established 
for the youth and adult population? ^ased on the differing needs and 
historical results for each populauon. Separate measures for adult 
welfare recipients reflect the long-run goal of reduced welfare depen- 
dency. In addition, an entered employment rate measure will apply to 
State activities and substate area activities under the JTPA Title III 
Dislocated Worker Program. Brief definitions of the seven measures 
are given below. 

Adult Measures • Adult Follow-up Employment Rate — the number of adult respon- 
dents who were employed (full-time or part-time) during the 13th 
full calendar week after termination as a percentage of the total 
number of all respondents (terminees who completed follow-up 
intei-views). 

• AduL; Average Weekly Eamings at Follov.'-up — total gross weekly 
earnings for all adult responden^;s who were employed during the 
13th full calendar week after termination, divided by the total 
number of adult respondents employed at the time of follow-up. 

Welfare Measures • Welfare Follow-up Employment Rate — the nuniber of adult welfare 

respondents who were employed (full-time or part-time) during the 
13th full calendar week after tennination as a percentage of the 
total number of adult welfare respondents (terminees who com- 
pleted foilow-up interviews). 

• Welfare Weekly Earnings at Follow-up — total gross weekly earn- 
ings for all adult welfare respondents who were employed during 
the 13th full calendar week after termination, divided by the total 
number of adult welfare respondents employed at the time of follrw- 
up. 

Youth Entered Employment Rale — the number of youth who en- 
tered employment at termination divided by the total number of 
youth who terminated, excludingboth those potential dropouts why 
are reported as having remained in school and dropouts who are 
reported as having returned to school. 

Youth Employability Enhancement Rate — the number of youth 
who attained one of the employability enhancements at termina- 
tion, whether or not they also obtained a job, divided by the total 
number of youth who terminated. Youth Employability Enhance- 
ments include: 

a) Attained two or more PIC-recognized youth employment compe- 
tencies 

b) Completed major level of education following participation of at 
least 90 calendar days or 200 hours in a JTPA activity 

c) Entered and retained for at least 90 calendar days or 200 hours 
in non-Title II training or received a certification of occupational 
skill attainment 
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Youth Measures 
(See Note) 



d) Returned to and retained in full-time school for one semester or 
at least 120 calendar days (dropouts only), attained a basic or 
job-specific skill, and made satisfactory progress 

e) Remained in school for one semester or at least 120 calendar 
days (youth at risk of dropping out of school), attained a basic or 
job specific skill, and made satisfactory progress. 

NOTE: Youth teiminees who remain in/return to school and enter 
employment will be reported on the JTPA Annual Status Report as 
"Also Attained Any Adult/Y outh Employability Enhancement." Such 
terminees will not be excluded from the termination pool reflected in 
the denominator in the calculation of the Youth Entered Employment 
Rate. 

For 14-15 year olds, the acceptable competencies will be basic skills or 
pre-employment/work maturity. 

Dislocated Worker (Title III) • Entered Employment Rate — the number of individuals who en- 
tered employr.>8nt at termination, excluding those who wc re re- 
called or retained by their original employers after receipt of a layoff 
notice, divided by ths total number of individual? who terminated, 
excluding those who were recalled or retained by their original 
employers after having received a layoff notice. 

Additional Performance The Department will continue to support regression modeling for 
Measures seven prior measures which the Governor may use as individual state 
policies dictate. The two cost measures may not be used for incentive 
purposes. They are: 

- Adult Cost per Entered Employment 

- Youth Cost per Positive Termination 

For Title III, there is a regression model for an optional goal: 

- Average Wage at Placement 

Performance Standards The Secretary of Labor has established national numerical perfor- 
mance standards for each of the six core Title IIA performance 
measures based on an analysis of prior accomplishments. The stan- 
dards are generally set at a level that approximately 75% of the SDAs 
are expected to exceed. Earnings, however, have been adjusted to 
reflect increased minimr.m wage races. Because program data for 
Title HI is not yet available, the numerical standard is unchanged 
from the previous yoar. The national performance standards for 
Program Year 1990 (July 1, 1990-June 30, 1991) are as follows: 
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Adult Standards Adult FoIIow-up Employment Rate 62% 

Adult Weekly Earnings at FoIIow-up $204 

Welfare Standards Welfare Follow-up Employment Rate SV/o 

Welfare Weekly Earnings at FoIIow-up $182 

Youth Standards Youth Entered Employment Rate 45% 

Youth Employability Enhancement Rate 33% 

Dislocated Worker (Title III) Entered Employment Rate 64% 



Local The national standards may be adjusted based on local conditions. The 
Adjustment Factors factors that are considered in this adjustment fall into two categories: 

the characteristics of participants who terminate from the program 
and local economic conditions. 

Characteristics of The following characteristics were found to affect performance. They 
Terminees are used in the adjustment model for the core standards or optional 
goals: 



1. 


Female 


2. 


Age 14-15 


3. 


Age 29 and under 


4. 


Age 30-54 


5. 


Age 55+ 


6. 


Student 


7. 


Dropout 


8. 


Less than High School 


9. 


Post High School Atv.-indee 


10. 


Reading below 7th grade level 


11. 


Black 


12. 


Other minorities 


13. 


All minorities 


14. 


Offender 


15. 


Unemployed 15+ weeks 


16. 


Not in labor force 


17. 


Handicapped 


18. 


Unemployment compensation claimant 


19. 


AFBC recipient 


20. 


Long-term AFDC recipient 


21. 


GA/RCA recipient 


22. 


Previous wage 
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Economic/Local Factors Seven additional factors which reflect characteristics of the local area 

are included in the model. They are: 



23. Unemployment rate 

24. Percent of families below the poverty level 

25. Annual earnings in retial/wholesale trade 

26. Population density 

27. Employee/resident worker ratio 

28. Three year growth rate in real annual earnings in retail/ 
wholesale trade 

29. Employment in manufacturing, mining and agriculture 

Noc every factor is used to establish every standard. For example, the 
percentage of 14 and 15 year olds is used only for the youth measures; 
aiid the percent who are ages 30 to 54 is used only for the adult 
standards. 

The state may establish an SDA's performance standards using the 
Secretary's National standards adjustment methodology or its own 
adjustment methodology established within the parameters set by 
USDOL. Adjustment methodology takes into consideration local SDA 
conditions and calculates a numerical performance level for each of 
the measures. SDAs must plan and operate programs to meet the 
established standards or seek adjustments to the state-established 
standards when local conditions warrant. 

The remainder of this guide provides an approach and some tools that 
v^U enable SD A/PIC siafFto use the performance standards as a "point 
of departure" for improving planning, management, and follow-up 
practices. 
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Section II: Identification of Local Performance 



In order to begin a serious examination of performance, an SDA must 
have accurate, complete program information available on a continu- 
ous basis. A sound management information system is essential. 
Without these resources, a manager does not have the ability to 
determine the status of current activities and cannot begin the process 
of improving performance. This first level of review, which includes 
calculating current performance, baseline program review, and trend 
anct comparison analysis, allows the SDA to conduct a comprehensive 
analysis of overall performance. 

Calculating Current Calculating your current performance enables you to take a snapshot 
Performance of local SDA performance and compare it to the six core JTPA Title IIA 
performance standards, and the Dislocated Worker standard, either 
as approved in your Job Training Plan or as adjusted based on actual 
experience. This concept is a bit tricky. Because the performance 
standards for a local area are adjusted based on the characteristics of 
terminees and characteristics of the local area, the standards will 
change depending on who is terminated and on local economic factors, 
such as the unemployment rate. The performance standards con- 
tained in an SDA's Job Training Plan will be based on planned service 
to the various target groups and economic factors as they are antici- 
pated ac the time the plan is done. It is most accurate to analyze your 
performance based on performance standards that have been ad- 
justed for actual performance. 

Worksheet 1, Performance Status Report, will provide a report on 
current actual performance standards. Under the "Actuals" section of 
the report, the adjusted Follow-up Employment Rate and Follow-Up 
Weekly Earnings refer to follow-up results after making adjustments 
fc"^ nonresponse bias. For SDAs that wish to include the adjusted 
standards. Worksheets 13, 14, 15 and 16 provide the methodology to 
generate adjusted standards. The Performance Status Report indi- 
cates how close you are to meeting your standards. The required 
information maybe available through existing systems either within 
your SDA or from the state. 

For Worksheet 1, the following information is required. 

1. Number of adults tenrinated 

2. Number of adults entered employment 

3. Average wage at placement for adults 

4. Total expenditures for adults 

5. Adjusted follow-up employment rate for adults 

6. Adjusted follow-up weekly earnings for adults 

7. Postprogram response rate for adults 

8. Number of adult welfare recipients terminated 



Number of adult welfare recipients entered emplo)niient 

10. Average wage at placement for adult welfare recipients 

11. Adjusted follow-up employment rate for adult welfare 
recipients 

12. Adjusted follow ip weekly earnings for adult welfare recipi- 
ents 

13. Postprogram response rate for adult welfare recipients 

14. Number of dislocated workers terminated 

15. Number of dislocated workers entered employment 

16. Average wage at placement for dislocated workers 

17. Total expenditures for dislocated workers 

18. Number of youth terminated 

19. Number of youth entered employment 

20. Number of youth who :'eturned to or remained in school 

21. Number of youth employability enhancements 

22. Number of youth positive terminations 

23. Total expenditures for youth 

24. Planned pei-formance standards as contained in the Job 
Training Plan or adjusted standards as calculated on 
Worksheets 13, 14, 15, and 16. 

If information on any category is not available, it is not possible to 
identify your current level of performance in relation to the perfor- 
mance standards. Adjustments to your Management Information 
System may be required. Worksheet 1 also contains three perfor- 
mance indicators, the entered employment rate for adults and adult 
welfare recipients, the cost per entered employment for adults and 
dislocated workers, and the cost per positive termination of youth. 
Worksheet 1, with these indicators and the performance standards, is 
designed to provide an ongoing analysis capability, allowing SDAs to 
increase the frequency of performance analysis beyond that currently 
provided by most state and local standardized reports. 
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WORKSHEET 1 



Report Period: 



Performance Status Repeirl 



ADULT WELFARE DISLOCATED YOUTH 



A 
C 
T 
U 
A 
L 
S 



1. Number Of Terminees 



I 

N 
D 

I 

c 

A 
T 
0 
R 

S 



S 
T 
A 
N 
D 
A 
R 
D 
S 



2. Number Entered Employment 



3. Number Returned/Remained in School 



4. Employablllty Enhancements 

5. Positive Terminations 

6. Average Wage at Placoment 

7. Total Expenditures ^ 

8. Adjust ed F/U Employment Rate 

9. Adjusto d F/U Weekly Earnings 
10. Postprogram Response Rate 



Entered Employment Rate 



Cost Per Entered Employmeni 



Cost Per Positive Termination 



Follow-Up 
Employment 
Rate 



Follow-Up 
Weekly 
E2rnings 



Entered 
Employment 
Rate 



Employability 
Enhancement 
Rate 



Actu?! Performance 




Expected Performance 



% of Standard 



Actual Performance 



Expected Performance 



% of Standard 



Actual Performance 



Expected Performance 



% of Standard 



Actual Performance 



Expected Performance 



% of Standard 





BetSeline Program Baseline program review helps identify probable causes of poor 
Review performanco through a comprehensive review of program outcomes 
and client characteristics. It begins the process of narrowing down the 
specific areas which influence performance by gathering data on 
program and subcontractor performance. 

The process begins with a review of total enrollments> terminations 
and expenditures. Worksheet 2 provides a suggested format for this 
review. This will give you a general idea of whether the program is 
progressing as planned. Deviations from plan in the areas of total 
served, youth service levels, total terminations and expenditures 
could have a dramatic impact on actual performance. 

The next two steps in the baseline program review process involve a 
detailed analysis of actual performance: first by target groups and 
then by prog^rams or contractors. Worksheets 3, 4 and 5 will enable an 
SDA to analyze, by performance standard. Title IIA adult, welfare and 
youth and Title III outcomes by teitninee characteristics. Worksheets 
6, 7 and 8 v/ill enable an SDA to analyze, by performance standard, 
adult, youth and dislocated worker outcomes by program or contrac- 
tor. Taken together, these reports will, in most cases, help isolate 
problem areas and provide a starting point for corrective action. This 
information will be useful when you are planning new strategies to 
improve performance since you will know what programs work best 
for what groups. They will also assist in designing new follow-up 
strategies. 

Although the type of analysis that can be done using Worksheets 3 
through 8 can be very useful in pinpointing problem areas, SDAs are 
cautioned not to apply the SDA's overall expected performance levels 
to all programs and contractors or to specific target groups. 

Programs must be looked at in the context of v "rying client needs and 
service strategies. For example, if an SDA has an average adult follow- 
up employment rate of 65% and average follow-up weekly earnings of 
$225, the SDA might be tempted to identify programs and target 
groups performing below this level as "problem areas." But, upon 
closer review, such programs may be exceeding model-based perfor- 
mance expectations based on the characteristics of the participants 
who terminated and the type of training being provided. Similarly, a 
program serving dropouts with a n employability enhancement rate of 
30% may not necessarily reduce overall performance. 

Not all activities will produce outcomes consistent with the overall 
expected performance levels established for an SDA — some will 
produce higher results and some lower. The challenge is to construct 
a job training system that, in total, exceeds overall established 
performance levels. 
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Analysis of Enrollments, Terminations and Expenditures 
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TITLE IIA 


12 MONTH 
PLANNED 


%0F 
TOTAL 


PLANNED 


%0F 
TOTAL 


/ICTUAL 


%0F 
TOTAL 


%0F 
PLANNED 
FOR PERIOD 


Enrollments (Tola!) 








Adult 
















Youth 
















Teiminations (Total) 








Adult 
















Youth 
















Expenditures (Total) 










Adult 
















Youth 
















Title III I 




ft 
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Enrollments 








\ 






Terminations 
















Expenditures 




t 
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Report Period: 
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Performance of Terminees by Charactenstic— AiluKt & Welfare (Title HA) 



ACTUAL PERFORr/lANCE 



CHARACTERISTIC 


ACTUAL TERMINEES 


ENTERED EMPLRATE 


AVERAGE WAGE 


FOLLOW-UP 
EMPLYMT 
RATE 


FOLLOW-UP 


NUMBER 


PERCENT 


CURRENT 


3 

MONTHS 
EARLIER 


CURRENT 


3 

MONTHS 
EARLIER 


AVERAGE 
WAGE 


WEEKLY 
EARNINGS 


Male 




















Female 




















Aqe 30-54 


. 








— — 








— — 


Aae 55+ 










- — 






— - 




White 




















Black 
















Other Minority 










— 








Dropout 








- — — 










Handicapped 




















Readii.g <7th Grade 




















UCCIaimaint 








' — 











— 


Unemployed 15+ Weel<s 












- — — 




— — — 




Not in Labor Force 


















AFDC 




















Long-Term AFDC 








GA/RCA 
















Total 


















Total Welfare 




















Other (Optional) 
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E?iCwORKSHEET3 



Report Period: 



Perforsnance of Terminees by Characteristic—Youth (Title IIA) 





ACTUAL TERMINEES 


ACTUAL PERFORMANCE 




NUMBER 


PERCENT 


CUTCDCn 

cn 1 cncu 
EMPLOYMENT RATE 


CsYirLUYADlUI Y 

ENHANCEMENT RATE 


Mal3 










Female 




•'-^ ' — 




— _ . 


Age 14-15 


■" — " 




— , — ^ 




Age 16-21 








_ _ 


White 


— ' •- ■ 






— — 


Black 











Other Minority 


., . 1... , . _ t 




— ' 


. ^ 


Dropout 


-Kt-L .1 , _ ^ 




— ^. — 


^ ^ 


Studant 






■ - - . 


, , . 


Post High School Attendee 






. — — 




Offender 










Reading <7th Grade 










Not in labor Force 










Total 










Other (Optional) 
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Perfoimance 



of Terminees by Characterisiic— Dislocatad Worker (Title III) 



unnriMu i CkMO i 11/ 


ACTUAL TERMINEES 


ACTUAL PERFORMANCE 


NUMBER 


PERCENT 


ENTERED 
EMPLOYMENT RATE 


AVERAGE WAGE 
AT PLACEMENT 


Male 








— - - - 


Female 






Age 29 and under 




— - 




Age 30-54 






Age 55+ 




• > 





White 




All Minorities 








Handicapped 










Less than High School 








Post High School Attendee 










UC Claimant 










linemployed 15+ Weeks 
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Analysis of Programs/Contractors— Adult 




PERFORMANCE DATA 


POST PROGRAM FOLLOW-UP 


PROGRAMS/CONTRACTS 


FNTFRPn 

EMPLOYMENT 
RATE 


WAGE AT 
PLACEMENT 


ruLLU w ur 
EMPLOYMENT 
RATE. 


rULLUWUr 

AVG. WEEKLY 
EARNINGS 


Occupational Training 

1. 

2. 

4. 

5. 
6. 










Total Occupational Training 






. .... 




Other Classroom Training 
1. Job Finding Skills 




• 






2 RpmAdial Pronram*; 
A. 
B. 










3. Special Programs 

A 
M. 

B. 
C. 










OJT 










Other 










Total All Programs 
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Analysis of Programs/Contractor&-:-Youth 




PERFORMANCE DATA 


PROGRAMS/CONTRACTS 


ENTERED EtyjPLOYMENT RATE 


EMPLOYABILITY ENHANCEMENT RATE 


Occupational Training 

1. 

2. 

3. 

4. 

5. 

6. 






Total Occupational Training 






Other Classroom Training 
1. Job Finding Skills 






2. Remedial Programs 
A. 
B. 






3. Special Programs 
A. 
B. 
C. 






OJT 






Other 






Total All Programs 







EWC WORKSHEET 7 Report Period: 



Analysis of Programs/Contractors— Dislocated 



PROGRAMS/CONTRACTS 


PERFORMANCE DATA 


ENTERED EMPLOYMENT RATE 


AVERAGE WAGE AT PLACEMENT 


Occupational Training 
1. 

2 
3. 
4. 
5. 
6. 






1 uidi uccupdiioiiai 1 raining 




— 


Other Classroom Training 
1. Job Finding Skills 






2. Remedial Programs 
A. 
B. 






3. Special Programs 
A. 
B. 
C. 






OJT 






Other 






Total All Programs 







WORKSHEETS 



Report Period: 



While analysis of the performance of specific program activities and 
subcontractors and the outcomes for specific client groups can be 
guided to a certain extent by the DOL model, DOL warns that a strict 
application of the model is these instances would be inappropriate. 

Trend Comparison Once you have calculated your current level of performance and 
and Analysis completed your baseline program review, you may want to conclude 
your overall performance analysis by examining performance trends 
and by comparing current performance with past SDA performance. 
In order to complete both the comparison and trend steps, you will 
need the following information: 

1. Prior year's performance standards (where available) 

2. Pei'formance reports (available from this section) 

3. Planned performance goals contained in the J ob Training Plan 
or as adjusted based on actual performance. 

Trend Analysis Trend analysis allows the SDA to visualize, on a regular basis, 
performance variations over a longer period of time. With this type of 
analysis, PIC members and SDA staff will easily be able to determine 
the direction in which the program is going. They can then take 
corrective action when a downward trend becomes apparent rather 
than waiting for performance to fall hopelessly below the standard. 

Performance may fluctuate at monthly or quarterly review points for 
legitimate reasons. Fluctuations may have a number of explanations: 
early terminations from a classroom training program are often 
negative; there may be funding delays; some programs coincide with 
the conventional school year. Trend analysis may help to pinpoint a 
problem area. It may indicate, for example, that your particular SDA 
has developed a trend of decreasing follow-up employment rates for a 
six month period. These trends would then be considered in analyzing 
SDA performance. 

Use Worksheets 9, 10 and H to maintain an ongoing recoi'd of trends 
for each of the performance standards groups — adult, welfare, youth 
and dislocated workers. Worksheet 12 can be used to track each 
standard on a yearly or more frequent basis and compare it with 
performance in previous years. 

The approach and reports detailed in this section complete the first 
level of performance review and shoiald provide sufficient information 
for SDAs to determine priority areas for further investigation and 
improvement. This baseline data should be carefully reviewed with 
loca^ policy makers and professional staff. Section'lII: Strategies for 
Improving Performance describes steps.that can be taken to improve 
performance— first the Oversight Response which helps pinpoint 
reasons for poor performance, the Planning Respon£ 3 which helps 
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identify new strategies for improved results, and the follow-up re- 
sponse which looks at possible postprogram issues. In almost all cases, 
an SDA will be involved in some combination of the Oversight 
Response, the Plannijig Response, and the Follow-Up Response. 
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WORKSHEETS 



Report Period: 



Adult & Welfare Trend Analysis 

Complete the bottom portion of the worksheet and 

graph the quarterly/monthly trends In performance standards. 


% OF STANDARD 


PERIOD ENDING 

/ / / 


PERIOD ENDING 

/ / / 


PERIOD ENDING 

/ / / 


PERIOD ENDING 

/ / / 


150% 
140% 
130% 
120% 
110% 

100% 

80% 

70% 
60% 
50% 
40% 










PERFORMANCE 
STANDARDS 


LEGEND 


% OF STANDARD 


Adult (F/U) 
Employment Rate 












Adult (F/U) 
Weekly Earnings 












Welfare (F/U) 
Employment Rate 








Welfare (F/U) 
Weekly Earnings 
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WORKSHEET 10 Report Period: 



1 

Youth Trend Analysis 

Complete the bottom portion of the worksheet and 

graph the quarterly/monthly trends in performance standards. 


% OF STANDARD 


PERIOD ENDING 

/ / / 


PERIOD ENDING 

/ / / 


PERIOD ENDiuc 

/ / / 


PERIOD ENDING 

/ / / 


15U% 










140% 










130% 










120% 
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100% 










90% 










80% 










•J no/ 

70% 










60% 










50% 










40% 










PERFORMANCE 
STANDARDS LEGEND 


% OF STANDARD 


Entered 
Empioyment 
Rate 










Employability 
Enhancement 
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WORKSHEET 11 



Report Period: 



Dislocated Worker Trend Analysis 

Complete the bottom portion of the worksheet and 

graph the quarterly/monthly trends in performance standards. 


% OF STANDARD 


PERIOD ENDING 

/ / / 


PERIOD ENDING 

/ / / 


PERIOD ENDING 

/ / / 


PERIOD ENDING 

/ / / 
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Report Period: 



Trend Analysis by Standard 



Select performance standard, complete the bottom 

portion of worksheet and graph the results Performance Standard 



% OF STANDARD 


PERIOD ENDING 

/ / / 


PERIOD ENDING 

/ / / 


PtiRIOD ENDING 
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Section Strategies for Improving Performance 



As noted throughout this Guide, the U.S. Department of Labor has 
significantly revised the Federal performance standards for Progi-am 
Yeat 90. The Department is now attempting to measure the longer- 
term success of those enrolled in JTPA. The objective of the changes 
in the perfomiance standards is to send a clear message to the JTPA 
system that resources and effort should be focused on those individu- 
als with multiple bamers to employment and those program compo- 
nents which benefit participants in the long-mn. This policy direction 
is consistent with the recommendations made by the Job Training 
Partnership Act Advisoiy Comrrittee in its Working Capital repoits 
and al^/O supports the Congi-essional intent in establishing JTPA. 

A second trend in evidence both in the Working Capital reports and in 
some of the language of the performance standards concerns coordi- 
nation with related systems to expand and increase the effectivene^ss 
of service deliveiy and improve outcomes. JTPA is no longer seen as 
a "stand-alone" system and increasingly exists within the context of s 
larger "workforce readiness" system in a local area. 

For SDAs that need to or want to improve performance, there are 
several approaches to take. The "Oversight Response" helps pinpoint 
the factors in your existing programs, systems and relationships with 
other agencies which may be causing perforaiance problems. The 
"Planning Response" helps identify new strategies for improving 
performance. The "Follow-up Response"looks atpossible postprogram 
issues and activities. All of these should be viewed as ways i 
validating or modifying the key policies established by the PIC and 
elected ofiicials. 

Oversight Response By conducting the baseline review and trend analysis, an SDA will be 

able to identify programs, contractors or target groups which seem to 
be less successful. The next step is to understand why performance 
may not be meeting expectations. The "Oversight Response" provides 
the SDA with a diagnostic tool to review and identify the factors that 
contribute to poor performance. 

In order for JTPA to function effectively at the local level, a complex 
and interrelated set of progi-am and administrative systems must be 
in place. These systems and procedures cover a broad range, including 
areas such as participant assessment, job development, contracting, 
MIS, supportive services, and classroom training. These syste. .is are 
operated either directly by the administrative entit" % in some cases, 
subcontracted to local service pro\dders. 
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These basic elements are classified in the following three categories: 



JTPA Services and Prpgrams — This is the most critical component 
of the employment and training system and includes the following 
activities: outreach, eligibility determination, assessment, employ- 
ability counseling, supportive services (including needs-based pay- 
ments), job development, classroom training; on-the-job training, try- 
out employment, work experience, and other specdal programs. 

Administrative and Management Svstems — The qu*)Jity and effec- 
tiveness of JTPA programs and services are directly determined by 
the basic management and administrative systems which shape and 
support them. The most critical elements here include: planning 
(including youth competencies), organizational structure and staff- 
ing, fiscal management, MISj contracting, and monitoring. 

Or ganizational Roles anoRelationshins — JTPA allows wide latitude 
in determining the roles and relationships formed both within local 
SDAs and with other agencies involved in workforce readiness pro- 
grams, such as Employment Semce, Welfare Department (through 
JOBS and other programs) and the various Education agencies. The 
scope and nature of these roles can have a profor.id influence on the 
quality and cost of the programs and services provided under JTPA. 
The critical areas here include: responsibilities of grant recipient and 
administrative entity, planning responsibilities, cooperative relation- 
ships with the other agencies involved in workforce readiness, and 
leveraging of non-JTPA funds. 

The Performance Status Reports, Baseline Program Review Reports^ 
and Trend and Compari son Analysi s Reports detaile d in Section II can 
provide you with an abundance of data which offer specific "access 
points" to program areas, contractors, or activities with performance 
problems. Large scale deviations from a given performance standard 
should be viewed by management as a flag of a possible problem. 
However, the performance standards model cannot tell the program 
manager exactly what the problem is. 

For exam'^le, suppose an SDA's follow-up employment rateis substan- 
tially beluw the standard. The manager would first identify both 
target gi'oupsi jd programs or contractors which deviate significantly 
from the standard. This could be determined from the baseline 
program review conducted in Section II. The manager would then 
consider whether this deviation is reasonable or not. If it is an 
occupational training program, what point in the training cycle does 
this represent? Doct this represent a significant "drop-out" rate firom 
the program? Because there is generally a high correlation between 
the follow-up employment rate and the entered employment rate, it is 
worth looking at the entered employment rate from both 3 months 
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earlier and the current period, to see whether theproblem was caused 
at the planning level, the conti^acting level or if there is a problem with 
cost. Thus, a review could be conducted of the planning and contract- 
ing systems of the SDA and the fiscal system of the specific programs 
or contractors. It is important to look at the entered employment rate 
and average wage at placement for the current period so that problems 
are identified aS quickly as possible. Thib simplified example does not 
address many important factors, but it does illustrate the type of 
branching analysis that can pinpoint the cause of performance prob- 
lems. 

Some SDAs may opt for a total review of all the suggested systems in 
order to ma:dmize performance rather than confine the review to 
programs identified as having problems in the baseline review. By 
conducting a complete review of all systems, the SDA has an opportu- 
nity to approach performance improvement in a more comprehensive 
manner, allowing for, in most cases, a one-time alignment of these 
primary systems. 

SDAs should use the Oversight Response Summary to identify the 
program, administrative, and organizational systems that may be 
causing poor performance. Since all systems may affect performance 
standards to some extent, SDAs should be familiar with the interre- 
lationship of the standards and concentrate their reviews on the 
primary problems. 

Once an SDA has determined which systems should be examined 
further, it should refer to the Systems Sunmiaries which provide 
guidance on both the areas to be reviewed and the method of review. 
Finally, appropriate corrective action plans should be formulated and 
implemented. 
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Oversight Response Summary 



JTPA SERVICES AND PROGRAMS 



MOST LIKELY TO INFLUENCE ENTERED 
EMPLOYMENT RATE, FOLLOW-UP 
EMPLOYMENT RATE A' 3D 
EMPLOYABILITY ENHANCEMENT RATE 



MOST LIKELY TO INFLUENCE 
FOLLOW-UP WEEKLY 
EARNINGS AND AVERAGE 
WAGEATPUCEMENT 



Services 

Eligibility Determination 

Assessment 

Counseling 

Needs-Based Payment System 
Supportive Service System 
Job Develop mentfiPla cement 



Programs 

SkiWocational Training 
Basic/Remedial Education 
Special Targeted Programs 
Work Experience 
OJT 

Youth Competency System 



Initial outreach and assessment 
(offering client most 
appropriate training) directly 
affect dropout rate. 

Ongoing counseling, needs- 
based payment and other 
supportive services affect 
retention. Job development is 
critical to the entered 
employment and follow-up 
employment rates. 

The qualit\' of training and its 

appropnateness to train 

directly affect these 

standards. 
The Youth Competency System 

will affect the Youth 

Standards. 



Job development/placement most 
directly affects these 
standards. 



The design and quality of the 
training and OJT affect the 
earnings of completers. 



Administration and 
Management Systems 
Planning (including Youth 

Competency System) 
Organization/Staffing 
Fiscal 
MIS 

Contracting 
Monitoring 



Organizational Roles and 
Relationships 
PIC-LEO Agreement 
Employment Service 

Agreement 
Coordination with: 

Welfare 

Education 

Economic Development 
Unemplo>7nent 
Insurance 



Planning in demand occupations 

is essential. 
Quality of youth competency 

system could have a major 

impact on employability 

enhancement rate. 
Accurate fiscal and MIS systems 

as well as clear contracts are 

required to demonstrate 

performance. 
Monitoring is essential to 

uncover problems on a timely 

basis. 



Problems may occur at the 
planning or contracting ?tage. 



Coordination with welfa^^e 
agencies is critical to the 
welfare standards. 

Coordination with education 
agencies is necessary for the 
employability enhancemen* 
rate. 



Relationships with the 
Employment Service, 
Economic Development 
Agencies and other job 
placement/creation agencies 
will affect these standards. 
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SYSTEMS SUMMARIES 



PROGRAMS AND SERVICES 



1. OUTREACH 

AREAS TO BE REVIEWED 
Organization and staffing 
Subrecipients' responsibilities 

Linkages with community agencies serving targeted groups 
Coordination with welfare and education agencies 
Specific outreach goals by target group and geographic area 
Outreach process — use of media, written material, personal contact 
METHOD OF REVIEW 

Examine organization charts, mission and function statements 

Review outreach goals by target group 

Interview staff 

Review sources of referrals 

Track characteristics of applicants, enrollees and terminees 

2. ELIGIBILITY DETERMINATION/SELECTION 

AREAS TO BE REVIEWED 
Intake/eligibility process 

Staffi'subrecipient coordinating agency responsibilities 
Length of time from application to enrollment 
Characteristics of eligibles compared to those enrolled 
Existence of specific goals 
METHOD OF REVIEW 
Review the eligibility system as described in JTPA 
Examine application/eligibility determination process 
Analyze characteristics of eligibles not enrolled 
Review intake forms for completeness and accuracy 



3. ASSESSMENT 

AREAS TO BE RE VIE Vv ED 

Design of assessment 

Administration of assessment 

Staffing and organization 

Quality of assessment 

Use of assessment data 

EDP development 

METHOD OF REVIEW 

Review assessment procedures and tools 

Sample assessment files 

Sample employability development plans 

Review skills and characteristics of successful completers 

Interview staff and participants 
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4. COUNSELING 

AREAS TO BE REVIEWED 

Organization and staffing 

Procedures for counseling throughout the Jl system and through related 
systems 

Use of the EDP as an ongoing tool 
Use of case management 
METHOD OF REVIEW 

Interview staff of JTPA and related agencies, if appropriate 
Interview clients 

Review client folders for progress reports and identification of needs/action 
plans 

Determine caseload trends 



5. NEEDS-BASED PAYMENT SYSTEM 

AREAS TO BE REVIEWED 

Methodology for needs-based payment determination 

Consistency of such payments for different target groups and activities 

Effect of payments and services on dropout rates and enrollment trends 

METiIOD OF REVIEW 

Review written procedures 

Track dropout rates and underenrollment trends 



6. SUPPORTIVE SERVICE SYSTEM 

AREAS TO BE REVIEWED 

Adequacy of other supportive services provided directly or throixgh agreement 

with other agencies: child care, medical, transpoiiation 
Method of identifying need for and arranging for these services 
MimOD OF REVIEW 
Review wiitten procedures 
Interview staff 
Interview clients 
Interview service agencies 



7. JOB DEVELOPMENT/PLACEMENT 

AREAS TO BE REVIEWED 
Organization and staffing 
Subrecipient responsibility 
Performance-based contract usage 

Coordination of job develoPnient with employment and economic development 
agencies 

Marketing plan and implementation 

METHOD OF REVIEW 

Interview job development and placement staff 

Review job orders and placement documentation 

Track specific vocational training i«ttivities with poor wage rates 
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Review coordination agreements with Employment Service and Economic 

Development 
Contact local employers 



8. SKILL/VOCATIONAL TRAINING 

AREAS TO BE REVIEWED 

Organization and structure of the course 

Instructional materials 

Staffing 

Instructor s flexibility and responsiveness 
Frequency and accuracy of trainee assessment 

Entered employment rate and wage at placement for trainees in courses 

METHOD OF REVIEW 

Review course curriculum 

Conduct site visits 

Interview instructor and trainees 

Review reports from training providers 

Discuss appropriateness of curriculum with local employers 



9. BASIC/REMEDIAL EDUCATION 

AREAS TO BE REVIEWED 
Organization and structure of the course 
Instructional materials 
Staffing 

Instructor's flexibility and responsiveness 

IVequency and accuracy of trainee assessment 

Completion by trainees of m ijor levels of education 

METHOD OF REVIEV/ 

Review course curriculum 

Conduct site visits 

Interview instructor and trainees 

Review reports from training providers 



10. SPECIAL TARGETED PROGRAMS 

AREAS TO BE REVIEWED 

Ability of targeted programs to increase participation of the 'Tiard to serve" 
Services and support provided to 'Tiard to .serve" populations 
METHOD OF REVIEW 

Perform a desktop review of each targeted program 

Review enrollments, positive terminations and placement data 

Compare enrollee profile to SDA goals for target groups 



11. WORK EXPERIENCE (YOUTH EXEMPLARY PROGRAMS) 

AREAS TO BE REVIEWED 

Organization and staffing 

Characteristics of clients served in this category 

Quality of training experience 
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METHOD OF REVIEW 

Review work experience agreements 

Conduct site visits 

Interview SDA staff responsible for monitoriRg worksites 
Interview participants 
Review timesheets 
Review progress reports 



AREAS TO BE REVIEWED 

Organization and staffing 

Written procedures describing specific responsibilities 

Linkages with other job development activity at subrecipient level 

Procedures for marketing 

OJT contract development 

METHOD OF REVIEW 

Review criteria, used to develop and approve OJT contracts 
Review of OJT contracts 
Conduct site visits 
Review completion rates 



AREAS TO BE REVIEWED 
Design of Youth Competency System 
Organization and staffing 

Characteristics of clients served by Youth Competency System 
Quality of Youth Competency System 
METHOD OF REVIEW 

Review Youth Competency System and method of documentation 
Observe classes 
Review completion rates 



Organization and staffing 
Plan preparation 
PIC involvement 

Planning process (including Youth Competency System) 

Procedures for tracking performance of subcontractx)rs 

Procedures for collecting and distributing ongoing performance information 

METHOD OF REVIEW 

Review written procedures 

Interview staff 

Review program evaluations 

Review PIC input through interviews or review of minutes 

Examine youth competency system procedures and monitor implementation 



12. OJT 



13. YOUTH COMPETENCY SYSTEM 



14. PLANNING 

AREAS TO BE REVIEWED 
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15. ORGANIZATION AND STAFFING 

AREAS TO BE REVIEWED 
Number and functions of staff 
Lines of authority 

Formal and informal mc»thods of exchanging information 
Staff training and development efforts 
Ratios ot service staff to client population 
METHOD OF REVIEW 
Review organizational charts and job descriptions 
Interview staff to validate lines of authority and information exchange 
procedures 

Examine the reasonableness or agency departments of units 
Analyze ratios of staff to clients for program and service systems 



16. FISCAL 

AREAS TO BE REVIEWED 

Staffmg and procedures 

Budgeting procedures and coov categories 

Accrual system, report preparation and distribution 

Subrecipient reporting system 

Timeliness of reports 

METHOD OF REVIEW 

Review back-up of total JTPA budget and subrecipient budgets 
Interview staff, review and validate monthly reports, accruals and report 
distrib»'Mon 

17. Mis 

AREAS TO BE REVIEWED 
Organization and staffmg 
Data collection 
Expertise of MIS staff 

Distribution of reports and validation process 
METHOD OF REVIEW 
Review written procedures 

Validate written procedures and definitions through interviews with staff 
Track sample reports to source documents 



-^8^ CONTRACTING 

AREAS TO BE REVIEWED 

Organization and staffmg 
Soliciting proposals, evaluating proposals 
Negotiating contracts, modifying contracts 
Evaluating contracts 

Staff expertise in federal, state, and local procurement regulations 
Clarity of request for proposals 
Specificity of contract goals and objectives 
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METHOD OP REVIEW 
Review written procedures 
Interview staff 

Track RFP process through contract execution and modification 
Review use of performance contracts 



Organization and staffing 
Monitoring procedures 

Frequency and depth of monitoring and evaluation 

Process for report distribution and corrective action 

Utilization of evaluation results in planning and contract modification 

METHOD OF ilEVIEW 

Review written procedures 

Interview staff and determine the level and frequency of monitoring 
Track reports through corrective action process 

Determine the extent to which evaluations are used in the planning process 



ORGANIZATION AND COORDINATION 



20. PIC-LEO AGREEMENT 



AREAS TO BE REVIEWED 

Clarity of roles and responsibilities 

Procedures for developing job training plan 

Level of involvement in policy guidance and oversight 

METHOD OF REVIEW 

Review written agreement 

Interview PIC members 

Review PIC meeting minutes 



SDA/Employment Service coordination agreement 
Staffing and organization 
Information exchange 
Client flow 

METHOD OF REVIEW 
Review written agreement 
Internew Job Service and SDA staff 
Track number and results of referrals 



19. MONITORING 

AREAS TO BE REVIt WED 



21. EMPLOYMENT SERVICE 

AREAS TO BE REVIEWED 
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22. WELFARE 

AHEAS TO REVIEWED 

SDA/Welfare aj^^reement 

Welfare recipient participation rates 

Information exchange and referrals 

Plan for JOBS implementation in local area 

METHOD OF REVIEW 

Review SDAAVelfare agreement 

Review \vritten procedures 

Interview staff of public assistance agencies and SDA 
Track numbers and results of referrals 
Review JOBS plan 



23. EDUCATION 

AREAS TO BE REVIEWED 

Coordination to define at risk youth and satisfactory progress 

Information exchange and referrals 

METHOD OF REVIEW 

Review correspondence or written agreements 

Interview appropriate education staff 

Track numbers and results of referrals 



24. ECONOMIC DEVELOPMENT 

AREAS TO BE REVIEWED 

Cooperative SDA/Econdmic Development agreements 

Procedures used to share local economic development and labor market 

information 
Information exchange: 

— Listing of companies planning to relocate to the SDA 

— Listing of companies anticipating expansion within the SDA 

— Listing of planned or actual plant closings or reductions in force within 

the SDA 
METHOD OF REVIEW 
Review written agreements 

Interview staff of economic development agencies and SDA 
Track results of any cooperative projects 



25. UNEMPLOYMENT INSURANCE 

AREAS TO BE REVIEWED 

Coordination agreement with the local Unemployment Insurance office 

Procedures for timely information exchange and referral of UI claimants 

METHOD OF REVIEW 

Review written procedures 

Interview appropriate UI and SDA staff 

Analyze number and results of UI referrals 
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Summary of Oversight Step 1: Identify specific performance measures needing improve- 
Response Approach ment through Performance Status Report. 



Step 2: Identifyprograms, contractors, orproblems with targetgroups 
which may be causing poor performance by using Worksheets 4, 5 6, 
7, 8 and 9. Make allowances for varying client needs and service 
strategies. 

Step 3: Identify probable systems causing poor performance either 
within the program or contractor or throughout the entire program 
using the Oversight Response Sumimary. 

Step 4: Use tihe Systems Summaries to conduct a review of each area 
identified abcve. 

Step 5: Design, implement, and follow up on corrective action recom- 
mendations. 

Planning Response In addition to correcting deficiencies identified through the "Over- 
sight Response," most SDAs engage in what is termed cae Tlanning 
(or Replanning) Response". After examining the results, they rethink 
key policy decisions and initiate new policies in such critical areas as: 

• Who will be served? 

JTPA funds are limited. SDAs must decide which groups of individu- 
als will receive priority v^thin JTPA and how JTPA will work with 
other workforce readiness programs which sei-ve similar populations. 
Individuals who are enrolled in another program may still be retained 
in JTPA as well. 

• What services and programs wall be provided? 

Taking into account the characteristics of the people who will be 
served and the availability of other training and remedial programs 
and services in the local area, SDAs determine the mix of programs 
and services that will be offered with JTPA funding and how financial 
resources "vill te allocated. 

• What arrangements should be negotiated with non-JTPA 
service providers? 

Increasingly, agi^AiCies are working together so that the pro^amB and 
services offered complement each other in such a way that the client 
receives the optimal combination of services. This could range from 
having existing education agencies provide remedial education while 
JTPA provides occupational training to having social service agencies 
provide childcare or health seivices which are needed to make a client 
truly employable. 
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• What will be acceptable performance standards? 
In addition to the minimum standards established by DOL and the 
governor, SDAs ma> ucvolop additional performance guidelines or 
request waivers of state standards when appropriate. 

Theperformance standards system, while establishingaccountability 
for such policy and planning decisions, does not usurp this important 
local policy making role. In fact, one of the strengths of the perfor- 
mance standards system is that it does not reward or penalize SDAs 
for these types of policy choices, but simply holds them accountable for 
achieving performance standards which reflect those choices. SDAs 
which allow performance standards to "drive^ the decisions in these 
key areas rather than consciously establishingpolicies will quickly get 
lost in a maze of variables and formulas at the expense of quality 
progi'ams. 

Local policy makers should, however, have a clear understanding of 
the mechanics of the performance standards system so that they can 
be aware of the performance-related implications of their decisions. 
The foil owing list shows that22 adjustment variables which affect the 
computation of an SDA's expected performance are directly influ- 
enced by the SDA's policy making and plar/ning decisions in the areas 
noted above. 

Adjustment Variables Influenced by SDA 

Who is Served 

1. Female 

2. Age 14-15 

3. Age 29 and under 

4. Age 30-54 

5. Age 55+ 

6. Student 

7. Dropout 

8. Less than High School 

9. Post High School Attendee 

10. Reading below 7th gi'ade level 

11. Black 

12. Other minority 

13. All minorities 

14. Offender 

15. Unemployed 15+ weeks 

16. Not in labor force 

17. Handicapped 

18. Unemployment compensation claimant 

19. AFDC recipient 

20. Long-term AFDC recipient 

21. GA/KCA recipient 

22. Previous Wage 
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Not Influenced hv SPA 

1. Unemployment rate 

2. Percent of families below the poverty level 

3. Annual earnings in retail/wholesale trade 

4. Population density 

5. Employee/resident worker ratio 

6. Three year growth rate in real annual earnings in retail/ 

wholesale trade 

7. Employment in manufacturing, mining and agriculture 

Moreover, as can be seen on the "Effect of Changing Variables on 
Performance Standards" chart, SDAs can determine precisely what 
impact each of the variables will have on their expected performance. 
The variables across from ea Ji f the performance standards are 
listed in order of their significance. For example, the variable which 
most significantly decreases t^ adult follow-up employment rate is 
the annual earnings in retail ^nd wholesale trade. The next most 
influential variable for decreasing this standard is the unemployment 
rate. Of those that can be influenced by the SDA, the most influential 
on this standard is the percent of dropouts. 



Effect of Changing Variables on Performance S2andard&-*PY90 

(Listed in order of significance of impact) 



A 
D 
U 
L 
T 



PERFORMANCE STANDAFiD 



Employment Rate- 
At Follow-Up 



RAISES STANDARDS 
IF INCREASED 



1. Three Year Growth Rate 



Weekly Earnings— 
At FoUow-Up 



1. Population Density 

2. Annual Eamininr^ RAV 
Trade 

3. Three Year Growth Rate 

4. UG Claimant 

5. 30-C4 Years Old 



LOWERS STANDARDS 
IF INCREASED 



1. Annual Eamf )gs RAV Trade 

2. Unemployment Rate 

3. Employment Mfg. Mng. Agr. 

4. Dropout 

5. Not in Labor Force 

6. GA/RCA Recipient 

7. Long Term AFDC 

8. Bhick 

9. Unemployed 15 Weeks or + 

10. AF.OC Recipient 

11. Other Minority 

12. Reading Below 7th Grade 



1. Family Below Poverty Level 

2. Unemployment Rate 

3. Handicapped 

4. Female 

5. Age 55 and Above 

6. Dropout 

7. Black 

8. Empl/Res Worker Ratio 

9. Unemployed 15 Weeks or + 



A 

D 
U 
L 
T 

W 
E 
L 
F 
A 
R 
E 



Employment Rate- 
At Follow-Up 



Weekly Earnings 
At Follow-Up 



1. Three Year Growth Rate 



1. Annual Earnings RAV Trade 

2. Unemployment Rate 

3. Employment Mfg. Mng, Agi 

4. Dropout 

5. GA/RCA Recipient 

6. Female 

7. Black 

8. Not in Labor Force 

9. Other Minority 

10. Unemployed 15 Weeks or + 

11. Long Term AFDC 



1. Annual Earnings RAV 

2. Population Density 

3. Three Year Growth Rate 

4. Post H.S. Attendee 



1. Family Below Poverty Level 
2» Female 

3. Dropout 

4. Age 30 and Above 

5. Black 

6. Reading Below 7th Grade 

7. Empl/Res Worker ratio 
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Effect of Changing Variables on Performance Standards— PY90 




(Listed in order of significance of impact) 








RAISES STANDARDS 




1 nWFR^ STANDARDS 




PERFORMANCE STANDARD 


IF INCREASED 




1<C 'UODCRCCn 

ir iNUncAocU 


Y 


Entered Employment 


1. Post H.S. Attendee 


i. 


Unemployment Rate 


0 


Rate 


2. Three Year Growth Rate 


2. 


14-15 Years Old 








3. 


Dropout 


U 






4. 


Offender 


T 
1 






5. 


Student 


H 






6. 


ivcauing ijCIOw tin vjrraQe 








7. 


Not in L^bor Force 








8. 


Black 




^mnlnvabilltv 


1. Uneniplo3nnent Rate 










2 Sfudent 








ndiw 


^ Mftf in T^lvkr f'*nrf*G 








cniereci ciiiipi./yiTi6ni 


1 TIP PHimnnf 


1. 


Unemployment Rate 


D 


Rate 


2. Pose U.S. Attendee 


2. 


Family Below Poverty Lovel 








3. 


Age 55 or Older 


1 

I 






4. 


All Minorities 


s 






5. 


Dropout 


1 






6. 


Handicapped 


mm 






7. 


Female 


n 






8. Unemployed 15 Weeks or + 


c 










A 










T 
1 










E 










D 


Average Wage at 


1. Previous Wa£ , 


1. 


Female 




Placement 


2. Annual Earning RAV Trade 


2. 


Age 55 or Older 






3. Three Year Growth Rate 


3. 


Unemployed 15 Weeks or + 






4. Post H.S. Attendee 


4. 


Handicapped 








5. 


Age 29 and Under 








6. 


UC Claiirnnt 








7. 


Dropout 








8. All Minorities 
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It is important to note that the impact of specific variations on the 
overall standard is not, in most cases, substantial. The cumulative 
effect, however, can significantly change the local SDA standards. 

Familiarity with the dynamics of the performance standards system 
will enable SDA staff to examme how local factors are shaping current 
expected performance and to determine what effect program modifi' 
cations would have on future expected performance levds. By follow- 
ing the "examining impact" instructions on the next page and using 
Worksheets 13, 14, 15 and 16, an SDA can identify the extent to which 
each factor is increasing or decreasing its expected performance 
levels. Users should also compare local values with extreme factor 
values detailed in Section G of the Department of Labor's Training 
and Employment Information Notice No. 3^ -o9, dated May 29, 1990. 

The value of this exercise is in understanding how local factors have 
shaped current expected performance levels, not in determining the 
extent to which the SDA has deviated from the national averages. 

Once an SDA has completed the type of analysis recommended in this 
guide, it may consider changes in program design or emphasis for the 
balance of the program year. The Baseline Program Pie view (^onducted 
in Section II provides valuable information for this replanning effort. 
Obviously any new program emphasis should be based on what has 
worked best, for what groups, and at what cost. By following the "dry 
run" instructions on the next page and using Worksheets 13, 14, 15 
and 16, SDAs will be able to determine precisely what effect these 
various policy and planning options would have on their expected 
performance levels. 



Follow-up Response For the first time, JTPA programs for adults are being measured 

based on what happens after these adults leave JTPA. This calls for 
program operators to look, for '.he first time perhaps, at such factors 
as: 



♦ What "auscs people to remain or not remain employed? 

• How does a participant's experience in JTPA relate to what happens 
afterwards? 



Although there has been found to be a high correlation between the 
entered employment rate and the follow-up employment rate, the fact 
remains that some people remain employed while others do not; some 
people get jobs on their 0\vn afler termination. Although follow-up 
data is required,Jt is not required for 100% of terminees, or even 
employed terminees, nor is a breakdown by program or target group 
required. In addition, follow-up is only required at the 13-week point. 
Some studies have found that employment status at the 13-weekpoint 
correlates well with longer-term employment, but this may or may not 
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be the case for particular individuals or groups. In short, in order to 
find out more about v;hy people are and are not employed after 
participation in JTPA, it may be necessary to either collect more 
follow-up data or conduct special surveys or studies periodically. It 
maybe useful to obtain information from both the former participants 
and their employers from time to tin 

Information shouldbe collected so that the SDA can be knowledgeable 
about what factors seem to lead to continued employment and what 
factors seem to be correlated with individuals leaving or being let go 
from their jobs. These factors may include: 

♦ the wage or salary itself 

♦ wage or salary increases, after how much time 

♦ the hours of work (e.g. day, night, shift-work) 

♦ fringe benefits, especially health benefits 

♦ ease of getting to and from work (transportation) 

♦ quality of supemsion received on the job 

♦ success of childcare arrangements 

♦ health of the individual 

♦ housing problems 

♦ other personal problems/situations 

♦ feeling of being appreciated 

♦ preparation for the job, in terms of skill 

♦ suitability of the work environment 

The information must be analyzed to see if there are trends or if there 
are particular employers that seem to be more and less successful with 
former participants. Depending on the information, the SDA may 
want to change some of its programs or services, institute new 
programs or services or even begin to provide or arrange for services 
to employers. These additional services can range from an earlier 
follow-up with individuals, for example a 4-week follow-up, to assis- 
tance to employers in training them in supervisory techniques. 
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UC Claimant 


7.9 
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' 


+.894 




Unemp1 15+ 


46.9 






-.081 
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Not in Lbr Force 


15J 






-.125 








Long Term AFDC 


10.0 




-.094 








AFDC 


257 




-.058 








GA/RCA 


5.0 






-.111 




-.933" 




Unemp Rate 


5.9 






-.377 






Annual Earn R/W 


13.5 




-.392 




+3.078 




Pop Density 
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+3.139 




Emp/Res Ratio 
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3 Yr Growth Rate 
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Net Effect of Variables 










National Departure Point 
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Local Expected Performance 
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PY 90 Peif ormance Standards Worksheet— Welfare 



Variables 


National 
Percent 


Actual 
Terms 


Percent 
of Terms 


Standards Calculations 




(Actual- National X Weight) 


Total Terms 1 1 
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Weekly Earning 
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Female 


79.5 






-.089 




-.642 


.Aqe_304_ 


50.6 










-.374 


Black 


32.5 






-.089 




-.315 




Other MInorltv 


10.8 






-.051 






— — «- 


Dropout 


29.4 




^ -.112 




-.380 


' — — — M. 


Post H.S. 


20.4 
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Reading < 7th Gr 


•30.0 






.... 
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Long Term AFDC 


33.0 
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GA/RCA 


12.9 
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Unemp! 15+ 


56.8 
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Not in Lbr Force 


22.8 






-.086 








Unemp Rate 


5.9 






-.547 








Annua! Earn R/W 


13.5 






-.754 




+3.646 




Pop Density 


.7 










+3.265 




Fam Below Pov 


97 










-.773 




3 Yr Growth Rate 


1.1 






+.347 




+1.786 




Emp Mfg/Mng/Agr 


23.4 






-.262 








Emp/Res Ratio 


98.1 










-.195 






















































Net Effect of Variables 










National Departure Point 


51.0 


182.00 


Local Expected Performance 
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PY 90 Performance Standards Worksheet--Youth 
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National Departure Point 




45.0 




33.0 
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PY 90 Performance Standards Worksheet-HQislocated 



Variables 


National 
Percent 


Actual 
Terms 


Percent 
of Terms 


standards Calculations 




(Actual ' National X Weigtit) 


Total Tarms I | 








Employment Rate 


Wage at Placement 
































Female 


40,0 




-.019 




-.0101 




Aqe 29 and under 


23.0 










-.0050 




Aqe 55+ 


8.0 






-.110 




-.0079 




All Minorities 


29.0 




-.078 




-.0028 




Dropout 


16.0 




-.078 




-.0038 




Post H.S. 


31.0 




+.033 




+.0051 




Handicapped 


3.0 






-.052 




-.0072 




UC Claimant 
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+.035 
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42.0 






-.018 




-.0076 




Previous Waqe 
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Unemp Rate 
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Net Effect of Variables 










National Departure Point 


64.0 




Local Expected Performance 
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Section IV: Conclusion 



Clearly, the Performance St^jadards system has evolved over the last 
eight years. The performance measures and standards currently in 
use reflect the recommendations made in two important reports, both 
issued in 1989. 

The first study, "Evaluation of the Effects of JTPA Performance 
Standards on Clients, Services and Costs" was conducted by the 
National Commission for Employment Policy. The study concluded 
that federal and state policies related to performance standards have 
both intended and unintended effects on client mix and level of service. 
The study recommended certain revisions to eliminate the unin- 
tended effects of these policies. Overall, the study found the impact of 
the standards to be positive, and consistent with congi'essional intent. 

The second report, "Working Capital: JTPA Investments for the 90's", 
was issued by the JTPA Advisory Committee The report recom- 
mended significant revisions to the performance standards system, 
many of which have been made for PY 90. These include the elimina- 
tion of cost standards and the reduction in the number of standards. 

What does the future hold? The policy directions for JTPA certainly 
involve: 

• Targeting those most at risk. Information is now being collected on 
the number of individuals who lack a significant work history, who 
are horT>"^^ss, and who have multiple barriers to employment. In 
additior Governors may not use cost standards as a basis for 
making incentive awards. 

• An emphasis on training investments that will have an impact in 
the long mn. The standards for adults involve only follow-up 
measures. Employability enhancements for youth are equal in 
bLatus to job placements. Information will be collected on the 
number of weeks in training. 

• Promoting skill attainment. New information will be reported for 
adults and youth on achievement of a major level of education and 
attainment of basii education and occupational skills. Dropout 
prevention is recognized as a valid outcome for youth. Several of the 
employability enhancements for youth include length of participa- 
tion requirements as a means of validating the enhancement. 

• Fostering coordination. The JTPA system must v/ork with the 
educationsystemtodefine "at-risk" youth and"satisfactoryprogress" 
in school. Participants can remain enrolled in JTPA while enrolled 
in non-JTPA-funded training. 
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While the focus and mechanics of the performance standards -ysf''-^ 
will change, performance management will continue as an integ^.al 
part of public job training policy. 

Performance standards are not "the" solution to effective JTPA 
planning and managenxent but rather one of oeveral tools available to 
administrators to improve performance. The c)iallenge to local policy 
makers is to use performance standards to guide the development of 
programs which will meet the needs of the eligible population and 
business community. 

In order to work effectively with performance standards, an SD A must 
have a clear vision of its role in the community. The key policy issues 
related to who is served, with what types of JTPA programs and 
services, in conjunction with what other agencies, are important 
building blocks in establishing the foundation for planning and 
management decisions. Since the performance standards model ad- 
justs an SDA's standards according to who is sei-ved, the degree of 
difficulty in meeting the standards should be the same, regardless of 
who is sei-ved. For local policy makers who fail to establish this 
foundation, performance standards can become a "numbers game" 
unrelated to the mission and purpose of JTPA in a community. With 
a proper foundation, perfonnance standards can assist policy makers 
and progi^am operators to have an effective program. 
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